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Report  Cards  for  Health  Care 

Is  Anyone  Checking  Them? 


hanges  in  health  care  organization  and  financing  in 
the  United  States  have  increased  consumers’  concerns 
about  the  quality  of  the  health  care  they  receive.  Among 
national  efforts  to  improve  the  quality  of  health  care,  perhaps 
none  has  been  as  prominent  as  the  movement  to  evaluate 
health  care  quality  and  to  disclose  the  results  of  that  evaluation 
publicly,  usually  in  the  form  of  “report  cards,”  “provider  pro¬ 
files,”  or  “consumer  reports,”  sometimes  communicated  via  the 
World  Wide  Web.  The  purpose  of  public  disclosure  of  infor¬ 
mation  on  quality  is  twofold:  to  facilitate  informed  choice  and 
to  stimulate  quality  improvement.  Researchers  at  RAND,  who 
have  been  among  those  involved  in  designing  health  plan  per¬ 
formance  evaluation  and  reporting  systems  for  several  of  the 
nations  largest  employers,  have  begun  evaluating  the  effects  of 
those  systems.  While  it  was  generally  assumed  by  health  policy 
planners  that  consumers  would  welcome  access  to  these  report 
cards,  the  existing  research  seems  to  suggest  that  this  is  not 
the  case.  In  order  to  learn  why  those  faced  with  health  care 
decisions  fail  to  use  comparative  performance  information  and 
what  can  be  done  to  increase  its  use,  we  have  examined 

•  how  information  for  report  cards  is  gathered  and  organized; 

•  who  uses  them  and  who  does  not; 

•  why  people  don’t  use  report  cards;  and 

•  what  might  be  done  to  make  them  more  user  friendly. 

Health  care  report  cards  are  constructed  from  information 
gathered  from  several  national  surveys  of  health  care  quality. 
Contrary  to  expectations,  these  report  cards  appear  to  affect  the 
health  care  decisionmaking  of  large  health  care  providers  more 
than  that  of  individual  consumers.  We  have  learned  through 
our  own  experience  and  that  of  others  that  the  way  report  cards 
organize  and  present  their  evaluations  strongly  influences 
whether  the  report  cards  are  used.  Moreover,  a  few  simple  tech¬ 
niques  of  effective  communication  can  improve  their  usability 


How  Are  Health  Care  Report  Cards  Created? 


Over  the  past  decade,  collection  and  release  of  comparative 
data  on  hospitals,  health  plans,  and  individual  physicians  have 
been  increasing,  a  reflection  of  the  increasing  demand  for 
disclosure  and  accountability  in  all  walks  of  life.  The  informa¬ 
tion  provided  includes  reputation,  mortality  rates  (for  specific 
procedures),  accreditation  status,  and  results  of  some  annual 
surveys  by  organizations  that  evaluate  health  care  quality. 

Providing  such  information  to  the  public  requires  the 
systematic  collection  and  evaluation  of  data  on  quality  of  care. 
These  evaluations  include  comparisons  of  performance  over 
time,  among  providers,  and  against  established  standards. 
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The  construction  of  all  health  care  report  cards  involves 
a  similar  set  of  steps.  The  first  step  is  to  build  a  database  of 
standardized  performance  measures.  Two  of  the  key  sets  of 
measures  about  health  plan  performance,  for  example,  are  the 
Health  Plan  Employer  Data  and  Information  Set  (HEDIS) 
and  the  Consumer  Assessments  of  Health  Plans  Survey 
(CAHPS).  HEDIS  was  designed  and  implemented  by  the 
National  Committee  for  Quality  Assurance  (NCQA),  a  non¬ 
profit  organization  that  is  the  nations  leading  accreditor  of 
health  plans;  CAHPS  was  designed  by  leading  researchers  at 
RAND,  the  Research  Triangle  Institute,  and  Harvard  Univer¬ 
sity  with  support  from  the  Agency  for  Healthcare  Research 
and  Quality.  Health  plans  supply  HEDIS  data  to  NCQA, 
whereas  third-party  vendors  collect  CAHPS  data  from  health 
plan  enrollces.  Table  1  shows  the  kinds  of  information  these 
data  sets  capture:  evaluations  of  the  quality  of  care  processes 
(based  on  indicators  of  effectiveness  in  preventing  and  treating 
disease),  access  to  care,  treatment  outcome,  and  patient  satis¬ 
faction  (using  the  results  of  CAHPS). 

The  second  step  in  constructing  a  report  card  is  to  orga¬ 
nize  the  information  to  be  communicated  into  a  hierarchy  of 
categories  called  a  framework.  People  tend  to  remember  and  use 
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Provider  Type 

Type  of  Information 

Examples 

Hospital 

Overall  mortality 

HCFA  ! 

■ 

Mortality  by  specific 
procedure 

New  York;  Pennsylvania; 
Pacific  Business  Group 
on  Health  (PBGH)  } 

Mortality  by  specific 
disease 

California  Hospital  Out¬ 
comes  Project;  Cleveland 
Quality  Choice 

Accreditation 

Joint  Commission  on  the 
Accreditation  of  Health¬ 
care  Organizations  (JCAHO) 

Rankings  by  specialty  area 

U.S.  News  &  World  Report 

|  Health  plan 

Process  of  care 

NCQA  (HEDIS) 

Patient  experience  with  care 

NCQA  (CAHPS) 

Accreditation 

NCQA 

Categorical  rankings 

CARS,  U.S.  News  &  World  ; 
Report 

Medical  group 

Process  of  care 

PBGH 

Patient  experience  with  care 

PBGH 

Physician 

Mortality  rate  for  coronary 
artery  bypass  graft  surgery 

New  York;  Pennsylvania 

Source:  Adapted  from  McGlynn  and  Adams,  2001 . 


information  more  easily  when  it  is  presented  in  a  small  number 
of  bits,  so  organizing  the  vast  array  of  evaluation  data  into  a 
small  number  of  categories  increases  the  usability  of  report  cards. 

The  final  steps  in  the  process  consist  of  analyzing  and 
summarizing  the  data.  In  order  to  make  meaningful  compar¬ 
isons,  researchers  must  devise  methods  to  estimate  the  values 
for  missing  data,  adjust  outcome  data  for  the  prior  health  status 
of  the  patients  included  (a  technique  known  as  risk  adjustment ), 
and  evaluate  various  aspects  of  care  on  similar  scales. 

The  CARS  and  HCFA  Experiences 

In  1998,  we  assisted  the  “Big  Three”  U.S.  automobile  manu¬ 
facturers  and  the  United  Auto  Workers  (UAW)  in  creating  a 
Coordinated  Autos/UAW  Reporting  System  (CARS)  for  eval¬ 
uating  the  performance  of  health  plans  that  contracted  with 
one  or  more  auto  manufacturers.  Using  a  similar  process,  we 
demonstrated  how  a  set  of  performance  reports  designed  for 
the  Health  Care  Financing  Administration  (HCFA)  could 
help  Medicare  beneficiaries  choose  among  health  plans.  When 
we  reviewed  the  methods  used  by  others  to  summarize  data 
for  reporting  systems,  we  discovered  wide  variations  in  the 
methods  used  to  summarize  performance  data.  Although  the 
methods  were  all  valid,  the  variations  led  to  significantly  dif¬ 
ferent  conclusions  from  the  same  data.  This  finding  raises  the 
concern  that  if  multiple  report  cards  are  issued  in  the  same 
geographic  area,  their  evaluations  may  conflict,  confusing  the 
public  and  leading  to  mistrust  of  the  entire  process.  Develop¬ 
ing  a  national  consensus  on  reporting  strategies  would  decrease 
the  likelihood  that  consumers  would  be  presented  with  con¬ 
flicting  information  about  the  same  plans  or  providers  and 
would  increase  the  likelihood  that  people  would  trust  and  use 
the  information. 

What  Have  We  Learned  About  How  People  and 
Organizations  Use  Report  Cards? 

Given  the  now  widespread  existence  of  health  care  report 
cards,  what  is  known  about  who  uses  them,  how  they  are  used, 
and  their  effectiveness?  In  collaboration  with  researchers  in  the 
United  Kingdom,  we  reviewed  all  published  evaluations  of 
U.S.  reporting  systems.  We  discovered  that  evaluation  of  report 
cards  has  not  kept  pace  with  their  development.  Although 
there  is  now  a  major  industry  centered  around  the  collection, 
analysis,  and  publication  of  quality  data,  attempts  to  evaluate 
the  effectiveness  of  these  efforts  have  been  sparse  and  difficult 
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to  conduct.  Further,  because  no  clear  agreement  exists  about 
their  actual  purpose,  evaluation  of  their  effectiveness  is  difficult. 

Report  Cards  Seem  to  Have  Little  Influence  Among 
Consumers  and  Physicians 

Although  evidence  from  surveys  and  focus  groups  suggests 
that  consumers  want  more  information  about  the  performance 
of  their  health  care  providers,  few  individuals  are  apparently 
influenced  by  the  information  in  their  decisionmaking.  Accord¬ 
ing  to  the  surveys,  consumers’  choice  of  hospitals  relied  more 
on  anecdotal  press  reports  of  adverse  events  than  on  the  com¬ 
parative  assessments  that  were  available.  Only  one  study  appar¬ 
ently  showed  any  significant  effect  of  public  disclosure  on 
consumer  decisionmaking.  This  study  evaluated  the  effects  of 
a  New  York  state  reporting  system  for  physician  and  hospital 
performance  on  coronary  artery  bypass  graft  (CABG)  surgery 
mortality  and  choice  of  physicians  and  hospitals  in  the  early 
1990s.  CABG  is  one  of  the  primary  cardiac  revascularization 
procedures  used  to  treat  coronary  artery  disease.  During  the 
study  period,  actual  and  risk-adjusted  mortality  rates  decreased 
significantly.  Those  physicians  and  hospitals  that  performed  the 
surgery  more  frequently  and  with  lower  mortality  experienced 
an  increase  in  their  market  share  compared  with  lower-volume, 
higher-mortality  providers. 

Large-scale  purchasers  of  corporate  health  plans  are  no  more 
likely  than  individual  consumers  to  use  report  cards  in  their 
decisionmaking.  Only  a  small  proportion  of  purchasers  reported 
even  using  accreditation  data  in  their  purchasing  decisions. 

Physicians  also  are  unlikely  to  use  publicly  available  infor¬ 
mation  as  the  basis  for  patient  referrals.  Although  physicians 
tend  to  consider  the  information  accurate,  they  seldom  share  it 
with  patients.  Moreover,  many  physicians  believe  that  public 
disclosure  of  performance  information  encourages  other  physi¬ 
cians  to  refuse  to  treat  those  patients  in  the  poorest  health. 

A  review  of  the  results  of  the  CABG  study  by  independent 
researchers  found  that  the  better-performing  providers  were 
not  turning  away  higher-risk  patients.  Nevertheless,  report 
cards  that  evaluate  physicians,  hospitals,  or  other  provider 
groups  based  on  treatment  outcomes  need  to  employ  appropri¬ 
ate  risk  adjustment  methods  to  ensure  that  the  prior  health 
status  of  individual  patients  has  been  taken  into  account. 

Health  Provider  Organizations  Have  Responded  Favorably 

In  contrast  to  consumers  and  physicians,  health  provider  orga¬ 
nizations  such  as  hospitals  seem  to  respond  to  comparative 
performance  data  like  that  provided  by  the  CABG  study.  Stud¬ 


ies  show  that  in  response  to  such  feedback,  hospitals  have 
demonstrated  improvement  in  performance  of  a  variety  of  pro¬ 
cedures  and  in  patient  outcomes.  For  example,  the  2001  NCQA 
report,  The  State  of  Managed  Care  Quality,  reported  improve¬ 
ments  from  1999  to  2000  in  a  number  of  quality  indicators  for 
managed  care  organizations.  These  indicators  included  com¬ 
prehensive  diabetes  care  (such  as  eye  exams  and  blood  sugar 
monitoring),  women’s  health  measures  (such  as  screening  for 
breast  and  cervical  cancer),  childhood  immunizations,  smoking 
cessation  advice,  and  several  measures  of  heart  disease  care. 

These  findings  suggest  that  future  work  should  focus  at 
least  in  part  on  optimizing  the  design  and  dissemination  of 
report  cards  for  health  provider  organizations.  However,  it  is 
not  yet  clear  whether  information  that  is  disclosed  to  the  pub¬ 
lic  has  a  greater  impact  on  organizational  performance  than 
information  that  is  shared  only  among  members  of  provider 
organizations.  The  assumption  has  been  that  hospitals  improve 
their  procedures  and  outcomes  in  response  to  consumer 
demand,  or  out  of  fear  that  consumers  will  react  to  negative 
publicity  by  taking  their  business  elsewhere.  If  that  is  the  case, 
the  effect  of  health  care  report  cards  on  the  one  group  that  has 
proven  most  responsive  to  them  might  begin  to  wane  when 
it  discovers  that  few  consumers,  individual  or  corporate,  are 
paying  attention  to  the  negative  publicity  that  the  report  cards 
generate.  Furthermore,  researchers  must  focus  their  attention 
on  just  why  it  is  that  consumers  fail  to  use  report  cards  and 
what  can  be  done  to  improve  their  use. 

What  Keeps  People  from  Using  Report  Cards? 


One  of  the  primary  reasons  for  the  public  disclosure  of  com¬ 
parative  health  care  assessment  is  the  belief  that  informed  con¬ 
sumers  would  be  empowered  to  make  better  decisions  about 
their  health  care  and  that  such  informed  choices  will  lead  to 
better  health  care.  In  surveys  and  focus  groups,  consumers  say 
they  want  more  information  on  which  to  base  decisions  about 
health  care.  But  why  do  they  fail  to  use  the  information  they 
receive?  Consumer  surveys  have  uncovered  several  reasons. 
Frequently,  consumers  report  having  difficulty  understanding 
the  information  they  are  given  because  of  its  technical  nature 
and  the  way  it  is  presented.  Consumers  also  report  that  the 
types  of  information  available  are  not  relevant  to  the  decisions 
they  need  to  make,  or  that  too  much  information  is  presented 
and  no  guidance  is  provided  on  how  to  sort  through  it.  Studies 
of  how  people  process  and  use  information  to  make  decisions 
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show  that  up  to  a  point,  people  s  confidence  in  their  decisions 
increases  with  the  amount  of  information  they  are  given  on 
which  to  base  those  decisions;  however,  past  that  point,  pro¬ 
viding  more  information  actually  causes  people’s  confidence 
to  wane. 

Additional  concerns  are  that  the  information  is  not  avail¬ 
able  when  it  is  needed,  that  too  little  time  is  available  to  review 
the  material,  and  that  information  furnished  by  the  health  plans 
and  providers  themselves  may  not  be  trustworthy.  The  surveys 
show  that  individual  consumers  rely  instead  on  information 
from  family  and  friends  when  making  health  care  decisions. 

What  Can  Be  Done  to  Increase  the  Usability  of 
Report  Cards? 

Cognitive  psychologists  have  learned  a  great  deal  about  how 
people  acquire  and  use  new  information  and  about  the  effective¬ 
ness  of  various  communication  styles.  We  reviewed  a  number 
of  the  report  cards  that  were  available  to  assess  how  they  pre¬ 
sented  their  information  relative  to  what  we  know  about  effec¬ 
tive  communication  styles.  A  key  problem  for  many  of  these 
report  cards  appeared  to  be  poor  presentation  of  information. 
Because  the  use  of  health  care  report  cards  to  help  with  decision¬ 
making  is  a  voluntary  activity,  users  are  not  compelled  to  con¬ 
tinue  working  with  material  that  does  not  sustain  their  interest. 

We  identified  a  variety  of  ways  that  researchers  and 
authors  could  increase  the  usability  of  report  cards: 

Engage  the  reader  at  the  outset  by  providing  a  familiar  context 
for  the  information  being  presented.  Material  has  often  been  pre¬ 
sented  with  no  cues  about  its  organization  or  how  to  use  it;  the 
report  cards  sometimes  lacked  even  a  uniform  organizational 
structure.  Research  shows  that  new  innovations  take  time  to 
diffuse  into  the  mainstream.  Thus,  if  employers,  health  plan 
purchasers,  and  others  want  to  increase  the  usability  of  health 
care  report  cards  now,  effective  communications  techniques 
such  as  those  described  in  Table  2  will  need  to  be  adopted. 


I  •  Create  the  framework  of  the  reporting  system  by  identify¬ 
ing  the  general  categories  of  information  that  are  most 
j  important  to  consumers  (that  is,  use  a  top-down  approach),  j 

•  Provide  a  clear  context  up  front  for  the  information  being 
presented  and  the  reasons  for  its  presentation,  to  enable 

1  consumers  to  assimilate  the  information  into  what  they 
already  know  or  believe. 

1  .  .  ^ 

•  Organize  the  information  that  is  presented  to  the  consumer  t 

with  a  clear,  consistent  structure,  and  provide  cues  to  that 
structure  with  headings. 

:  f: 

•  Choose  a  structure  that  allows  consumers  to  access  the 

information  they  want  without  having  to  read  through  the 
entire  report.  t 

•  Present  information  in  more  than  one  way;  if  graphics  are 
used,  choose  those  most  appropriate  to  the  kind  of  infor¬ 
mation  being  presented  (that  is,  those  that  make  differences  j 
easy  to  perceive). 

Source:  Adapted  from  McGlynn,  Adams,  Hicks,  and  Klein,  1999. 

Provide  specific  educational  initiatives  for  target  audiences. 
Such  initiatives  may  include  use  of  the  mass  media  to  educate 
the  public,  use  of  continuing  professional  education  to  educate 
health  care  professionals,  and  release  of  the  data  as  part  of  an 
educational  package  aimed  at  providers  to  promote  quality 
improvement. 

Conduct  more  comprehensive  assessments  of  the  impact  of  public 
disclosure  of  information .  These  assessments  must  identify  who 
actually  uses  the  information  and  for  what  purposes.  Identify¬ 
ing  these  unknowns  will  allow  the  creators  of  health  care  report 
cards  to  tailor  the  messages  they  present  more  effectively  and 
to  reach  a  much  larger  segment  of  the  potential  users.  Our 
findings  thus  far  suggest  that  the  optimal  format  for  report 
cards  targeted  to  health  care  providers  and  plan  purchasers  will 
differ  considerably  from  the  format  that  will  appeal  most  to 
consumers  of  health  care. 


Abstracts  of  all  RAND  Health  publications  and  full  text  of  many  research  documents  can  be  found  on  the  RAND  Health  web  site  at  www.rand.org/health. 
RAND  is  a  nonprofit  institution  that  helps  improve  policy  and  decisionmaking  through  research  and  analysis.  RAND  Health  furthers  this  mission  by  working 
to  improve  health  and  health  care  systems  and  advance  understanding  of  how  the  organization  and  financing  of  care  affect  costs ,  quality ,  and  access.  RAND'*  is 
a  registered  trademark. 
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